Due to the evolution of the Internet and related technologies, transfer of data and sharing of information over the World Wide Web has become a very common phenomenon. The web provides a common interface for applications to interchange information, however, data exist in various forms and may be scattered in different platforms and locations. The challenge is to provide an ability to integrate the diverse, distributed data and information sources both within and outside an enterprise into a single coherent framework while meeting scalability, performance and business process requirements.
The plans, methods, and tools aimed at modernizing, consolidating, and coordinating the computer applications in an enterprise is referred to as Enterprise Application Integration (EAI). Typically, an enterprise has existing legacy applications and databases and wants to continue to use them while adding or migrating to a new set of applications that exploit the Internet, e-commerce, extranet, and other new technologies. EAI may involve developing a new total view of an enterprise's business and its applications, seeing how existing applications fit into the new model, and then devising ways to efficiently reuse what already exists while adding new applications and data.
EAI is one of the biggest challenges for large high-tech companies. A major component of the integration effort involves development of data interchange system and tools that can seamlessly transfer data among various systems within the organization. The primary goal of such systems and tools is to provide a generalized software infrastructure that can support various data sources, allow for consumption of data by downstream tools and define a process by which multiple systems can communicate using a supplier, consumer relationship. This needs to be accomplished taking into account co-existence of new and old (legacy) data sources, formats and systems already in place.
One example of an organization that needs data integration is a chip manufacturer that creates custom integrated circuits having specialized functions. To achieve the functions desired, the chips may be designed using software applications, such as Electronic Design Automation (EDA) software tools. The software applications are typically run on different computers may use and store different types of data. The chip manufacturer may have a need to integrate all of the design data, such that it can be retrieved for future analysis. Examples of the type of data that needs to be analyzed may include historical trends, design metrics, targets for process optimization, and so on. The problem is that not all applications have access to the data produced by applications running on different computers. And even where the computers are networked together as clients, the client applications are not compatible with each other in terms of how the data is formatted.
Enterprises, such as the example chip manufacturer, face many problems with respect to implementing enterprise level integration. One of the biggest problems is keeping abreast of the evolution of business applications. Critical business processes and systems evolve rapidly over time and hence there is major need for applications to be scalable. The various data requirements for applications change over time and hence the data interchange tools and methods needs to be generic and scalable. The example chip manufacturer, for instance, may need a way to integrate the data from the data sources in a manner such that the collection of data can evolve over time to include additional types of information from other sources, such as packaging and manufacturing data.
The second major problem with enterprise level data integration is that there is no standardized, modular, software toolkits that client applications can use out of the box for exporting and importing data across applications. There are some high-end third party solutions for this problem, but there are no low cost alternatives and they come with major deployment costs for bringing them online. Further compounding the problem is the fact that in some businesses, such as the example chip manufacturer, the people who must integrate the client applications with the third party tools are often design engineers, not programmers.
Also, the current solutions may provide a way of integrating existing data but there is no standard available for generating and transferring the data easily for applications to adopt. Although XML serves as a standard medium for transferring data from one system to another, it adds a lot of overhead for applications to implement XML specific data generation methods. Many application's developers may not have the knowledge to develop XML documents and the technology required to integrate the data. Even if the data exists in a transferable form, the issue of exporting the data into a data warehouse still exists.
Last, but not least, is that in current business environments, organizations do not have a lot of manpower, time, and material to invest in solving data integration problems. So any solution pursued needs to be cost effective, be able to be developed and deployed expeditiously and should minimize any impact on existing flows and methodologies.
There are some commercially available products built on web services architecture that attempt to address some of these problems. IBM's WebSphere Application Server provides a scalable transaction engine for dynamic e-business applications. The Open Services Infrastructure allows companies to deploy a core-operating environment that works as a reliable foundation capable of handling high volume secure transactions and Web services. Microsoft's BizTalk Server provides organizations with the server, tools, adapters, and vertical accelerators needed to integrate and automate their business. And TIBCO's web services solutions provide enterprise level integration and ability to create “Compound Web Services” that integrates web services.
Although each of the above-mentioned solutions addresses the problem of integrating data and communicating across applications, the existing solutions have several disadvantages. First, these solutions do not address the problem of how the data collected from the various data sources should be standardized and collected from the client applications. Second, all the above-mentioned solutions come with very high deployment costs and rely upon a business model based on professional turnkey services for integration. The cost of technology ramp-up of applications that need to make use of this is very high.
The third problem is that the overhead of integrating these solutions with individual applications is difficult because most applications have to be adjusted to support a Service Oriented Architecture (SOA), which these solutions expect. In addition, the overhead of integration is equally complex regardless of the scope and size of the applications.
Although Web services are deemed to provide a standard means of communication among different software applications, the learning curve of programmers to implement web services is very high. Also, organizations need to invest a considerable amount of both time and money to enable the use of web services, regardless of the size and complexity of the applications.
Accordingly, what is needed is an approved method and system for performing enterprise-level data integration. The present invention addresses such a need.